Controlling haptic feedback for enhancing navigation in a graphical environment

ABSTRACT

Method and apparatus for controlling haptic feedback to enhance navigation of a cursor or other controlled displayed object in a graphical environment. An interface device is capable of communicating with a computer running an application program and generating a graphical environment includes an actuator for outputting a haptic effect to a user of the interface device. A modulator modulates the magnitude of the haptic effect in relation to, in various embodiments, a velocity of the cursor or user manipulatable object; a rate of interaction of the cursor with graphical objects; or an amount of time that the cursor engages the graphical object.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of to the followingco-pending U.S. patent applications:

U.S. patent application Ser. No. 09/756,617, filed Jan. 8, 2001, whichis a continuation of U.S. patent application Ser. No. 08/571,606, nowU.S. Pat. No. 6,219,032, filed on Dec. 13, 1995;

U.S. patent application Ser. No. 09/903,209, filed Jul. 10, 2001, whichis a continuation of U.S. patent application Ser. No. 09/499,338, now6,259,382, filed on Feb. 4, 2000, which is a continuation of U.S. patentapplication Ser. No. 09/160,985, now U.S. Pat. No. 6,232,891, filed onSep. 24, 1998, which is a continuation of U.S. patent application Ser.No. 08/756,745, now U.S. Pat. No. 5,825,308, filed on Nov. 26, 1996; and

U.S. patent application Ser. No. 09/992,123, filed on Nov. 13, 2001,which is a continuation of U.S. patent application Ser. No. 09/590,856,now U.S. Pat. No. 6,317,116 filed on Jun. 8, 2000, which is acontinuation U.S. patent application Ser. No. 08/879,296, now U.S. Pat.No.6,078,308, filed on Jun. 18, 1997;

and this application claims the benefit of U.S. Provisional ApplicationNo. 60/262,286, filed Jan. 16, 2001, and entitled, “Controlling HapticFeedback During Graphical Image Navigation”;

all of these disclosures being incorporated herein by reference in theirentireties.

BACKGROUND OF THE INVENTION

The present invention relates to an interface device that allows a userto interface with a computer, and more particularly to a haptic feedbackinterface device allowing a user to interface with a graphicalenvironment displayed by a computer.

Computer systems are used extensively in many different industries toimplement many applications, such as word processing, data management,simulations, games, internet browsing, and other tasks. A computersystem typically displays a visual environment to a user on a displayscreen or other visual output device. Users can interact with thedisplayed environment to perform functions on the computer, e.g., play agame, operate an application program, experience a simulation, use acomputer aided design (CAD) system, etc. Such user interaction can beimplemented through the use of a human-computer interface device, suchas a joystick, mouse, trackball, steering wheel, knob, stylus andtablet, “joypad” button controller, foot pedal, yoke hand grip, or thelike, that is connected to the computer system controlling the displayedenvironment. The computer updates the environment in response to theuser's manipulation of a manipulatable object (“manipulandum”) such as ajoystick handle or mouse, and provides feedback to the user utilizingthe display screen.

One visual, graphical environment that is particularly common is agraphical user interface (GUI). Information within GUI's are presentedto users visibly and/or audibly, such as through a video monitor andsound card. Common GUI's include the Windows® operating system fromMicrosoft Corporation and the MacOS operating system from AppleComputer, Inc. These interfaces allows a user to graphically select andmanipulate functions of the operating system, of application programs,and of the computer by using an input device, such as a mouse,trackball, joystick, or the like. Other graphical computer environmentsare similar to GUI's. For example, graphical “pages” on the World WideWeb of the Internet communication network utilize features similar tothat of GUI's to select and operate particular functions. Some computeraided design system, such as autoCAD also provide graphicalpresentations to the user. A graphical environment may also comprise agame or simulation environment.

Several types of tasks are typically performed by a user in a graphicalenvironment. A cursor is often used to select graphical objects ormanipulate graphical objects, such as resizing or moving the objects.The user must navigate a cursor through the graphical objects in theenvironment to perform these tasks and to place the cursor in desiredlocations to perform other tasks. Thus, it is desirable to provide aninterface device and method that will provide improved haptic feedback,for example while navigating a graphical environment.

SUMMARY OF THE INVENTION

The inventions disclosed herein are directed to enhancing the navigationof a cursor or other controlled displayed object within a graphicalenvironment, particularly in relation to other graphical objectsdisplayed in the environment. Some inventions are related to thevelocity or rate of object interaction of the cursor, while others arerelated to duration of engagement of the cursor with other graphicalobjects.

More particularly, one aspect of the present inventions provides aninterface device capable of communicating with a computer running anapplication program and generating a graphical environment The interfacedevice includes a user manipulatable object capable of controlling themotion of a cursor displayed in the graphical environment and anactuator for outputting a haptic effect to a user of the interfacedevice. A modulator modulates the magnitude of the haptic effect inrelation to a velocity of the cursor, where the magnitude of the hapticeffect at a lower cursor velocity is greater than the magnitude of thehaptic effect at a higher cursor velocity. A haptic effect can be outputfor each of multiple graphical objects over which the cursor moves. Themagnitude can be modulated based on the cursor velocity using one ormore predetermined functions, including linear, step, and adaptivefunctions.

In another aspect of the present inventions, an interface device iscapable of communicating with a computer generating a graphicalenvironment and includes a user manipulatable object capable ofcontrolling the motion of a cursor, an actuator for outputting a hapticeffect to a user, and a modulator to modulate the magnitude of thehaptic effect in relation to a rate of interaction of the cursor withgraphical objects displayed in the graphical environment. The magnitudeof the haptic effect at a lower rate of interaction is greater than themagnitude of the haptic effect at a higher rate of interaction. One ormore functions can govern the modulation of the magnitude. Themodulation can include examining a number of graphical objectsencountered by the cursor in a predetermined period of time to determinethe rate of interaction; or, the modulation can be based on a timeelapsed from the cursor exiting one graphical object and interactingwith another graphical object.

In another aspect of the present inventions, a method for adjusting amagnitude of haptic effects associated with graphical objects displayedin a graphical environment provided by a computer includes determining ahaptic effect to be output by an actuator to a user of an interfacedevice, where the haptic effect initiated is based on an interaction ofa cursor with one of the graphical objects, and where the determining ofthe haptic effect includes determining a magnitude of the effect; andadjusting the determined magnitude based on a current velocity of thecursor in the graphical environment, the adjusting being performed afterthe haptic effect is determined.

In another aspect of the present invention, an interface device capableof communicating with a computer running an application program andgenerating a graphical environment includes a user manipulatable objectcapable of controlling the motion of a cursor and an actuator foroutputting a haptic effect to a user. A modulator modulates themagnitude of the haptic effect based on an amount of time that thecursor engages the graphical object, where the magnitude of the hapticeffect is reduced after a predetermined period of time. The modulatorcan modulate the magnitude based on a time function, where the timefunction provides a magnitude inversely proportional to an amount oftime that has lapsed since the cursor engaged the target.

The present inventions control haptic output of a haptic feedback deviceto assist the user in navigating a graphical environment such as a GUI,where haptic feedback is provided when it would be helpful to the userand is reduced at other times. The velocity-based, rate of interactionbased, and engagement time methods allow the user to coarsely positionthe cursor and navigate graphical objects without haptic sensationsinterfering in selection, positioning, and other navigational tasks.

These and other advantages of the present invention will become apparentto those skilled in the art upon a reading of the followingspecification of the invention and a study of the several figures of thedrawing.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a haptic feedback system suitable for usewith the present invention;

FIG. 2 is a diagrammatic illustration of a graphical user interfaceincluding graphical objects associated with haptic effects;

FIG. 3 is a diagrammatic illustration of a graphical object havingexternal and internal forces associated therewith;

FIGS. 4a and 4 b are diagrammatic illustrations of the use of isometriccontrol over the zooming of a view;

FIGS. 5a, 5 b, and 5 c are graphs illustrating different embodiments offunctions used to provide a strength of haptic effect based on avelocity of the cursor, according to the present inventions; and

FIGS. 6a and 6 b are graphs illustrating different embodiments offunctions used to provide a strength of haptic effect based on anengagement duration of the cursor with a target.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

The present invention relates to controlling haptic feedback during thenavigation of a graphical image in a graphical environment, for examplea graphical environment generated by a computer. Although the process isillustrated in the context of navigating a cursor or pointer on acomputer display, the present invention can be used while navigatingother graphical images or and should not be limited to the examplesprovided herein.

FIG. 1 is a block diagram illustrating a version of a system 10 of thepresent invention comprising a host computer 12 and an interface device14. The interface device comprises a user object 34 manipulatable by auser 22. As the user object 34 is manipulated, sensors 28 detect themanipulation and the manipulation is communicated to the host computerto, for example, control the positioning of a graphical image, such as acursor, on a display device 20. In one version, the interface device isa haptic feedback interface device capable of providing haptic feedbackto the user, for example through the user object 34. By haptic feedbackit is meant any feedback to the user 22 that involves the user's senseof touch. For example, the haptic feedback may comprise the applicationof a passive force, an active force, and/or a tactile sensation, as willbe described.

In the version shown in FIG. 1, host computer 12 may be a personalcomputer, such as an IBM-compatible or Macintosh personal computer, or aworkstation, such as a SUN or Silicon Graphics workstation. For example,the host computer 12 can be a personal computer which operates under theMS-DOS or Windows operating systems in conformance with an IBM PC ATstandard. Alternatively, host computer 12 can be one of a variety ofhome video game systems commonly connected to a television set, such assystems available from Nintendo, Sega, Sony, or Microsoft. In otherembodiments, host computer system 12 can be a “set top box” which can beused, for example, to provide interactive television functions to users,or other devices or appliances providing electronic functions to users.

In the described embodiment, host computer system 12 implements a hostapplication program with which a user 22 is interacting via peripheralsand interface device 14. For example, the host application program canbe a computer aided design or other graphic design program, an operatingsystem, a video game, a virtual reality simulation such as a medicalsimulation, a scientific analysis program, or other application programthat utilizes haptic feedback. The host application program may comprisean interactive graphical environment, such as a graphical user interface(GUI) to allow the user to input information to the program. Typically,the host application provides images to be displayed on a display outputdevice and/or outputs other feedback, such as auditory signals.

In the version illustrated, host computer 12 includes a host processor16, random access memory (RAM) (not shown), read-only memory (ROM) (notshown), input/output (I/O) electronics (not shown), a clock (not shown),a display device 20, and an audio output device 21. Host processor 16can include a variety of available microprocessors from Intel, AMD,Motorola, or other manufacturers. Processor 16 can be singlemicroprocessor chip, or can include multiple primary and/orco-processors. Processor 16 preferably retrieves and stores instructionsand other necessary data from RAM and ROM, as is well known to thoseskilled in the art. The clock may comprise a standard clock crystal orequivalent component to provide timing to electrical signals used byprocessor 16 and other components of the computer system. The clock isaccessible by host processor 16 in the control process of the presentinvention, as will be described. Display device 20 is coupled to hostprocessor 16 by suitable display drivers and can be used to displayimages generated by host computer system 12 or other computer systems.Display device 20 can be a display screen (LCD, plasma, CRT, etc.), 3-Dgoggles, projection device, or any other visual interface. In oneversion, the display device 20 displays a graphical user interface and agraphical image, such as a cursor or pointer, for interaction therewith.In another version, display device 20 displays images of a simulation orgame environment. Alternatively or additionally, other images, such asimages describing a point of view from a first-person perspective orimages describing a third-person perspective of objects, backgrounds,etc, can be displayed. Accordingly, a user 22 of the host computer 12and interface device 14 can receive visual feedback by viewing displaydevice 20. Audio output device 21, such as speakers, is preferablycoupled to host processor 16 via amplifiers, filters, and othercircuitry well known to those skilled in the art. Host processor 16outputs signals to speakers 21 to provide sound output to user 22 whenan “audio event” occurs during the implementation of the hostapplication program. Other types of peripherals can also be coupled tohost processor 16, such as storage devices (hard disk drive, CD ROMdrive, floppy disk drive, etc.), printers, and other input and outputdevices.

In the version shown, host computer 12 may receive sensor data or asensor signal via a bus 24 in communication with the interface device 14and other information. Processor 16 can receive data from bus 24 usingI/O electronics, and can use I/O electronics to control other peripheraldevices. Host computer 12 can also generate and output a signal, or“command”, to interface device 14 via bus 24. The signal may be relatedto a haptic effect to be output by the interface device 14 to the user22.

The host computer 12 may display graphical computer images. Thesecomputer images are related to logical software unit collections of dataand/or procedures that may be displayed as an image by computer 12 ondisplay screen 20, as is well known to those skilled in the art. Thegraphical image may interact with other graphical images within agraphical environment, and the interaction may be under the control of auser. For example, the graphical image may comprise a cursor or pointerthat interacts with other graphical images in a graphical userinterface, such as a Windows operating system. Alternatively oradditionally, a gaming image may interact within a gaming graphicalenvironment. For example, the graphical image may comprise athird-person view of a car. Alternatively, the graphical image may be atleast a portion or all of the graphical environment as viewed from afirst-person perspective, such as a displayed or simulated cockpit of anaircraft or a view from the cockpit of an aircraft.

In one version, the interface device 14 may coupled to the host computer12 by a bi-directional bus 24. The bi-directional bus sends signals ineither direction between host computer 12 and the interface device 14.Bi-directional bus 24 can be a serial interface bus, Universal SerialBus (USB), MIDI bus, Firewire bus, parallel bus or other protocol buswell known to those skilled in the art. A wireless communication link(e.g., radio or infrared) can alternatively be used.

Interface device 14 may include a local processor 26, sensors 28,actuators 30, a user object 34, optional sensor interface 36, anoptional actuator interface 38, and other optional input devices 39. Theinterface device 14 may comprise one or more housings for holding thesensors 28, actuators 30, local processor 26, and other relatedelectronic components, to which user object 34 is directly or indirectlycoupled. In the preferred embodiment, multiple interface devices 14 canbe coupled to a single host computer system 12 through bus 24 (ormultiple buses 24) so that multiple users can simultaneously interfacewith the host application program. In addition, multiple players caninteract in the host application program with multiple interface devices14 using networked host computers 12.

Local processor 26 may communicate with host 12 via bus 24 and may beincluded within a housing of interface device 14 to allow quickcommunication with other components of the interface device. Localprocessor 26 is considered “local” to interface device 14, e.g., atleast partially dedicated to haptic feedback and sensor I/O of interfacedevice 14. Processor 26 can be provided with software instructions towait for commands or requests from computer host 16, decode or parse thecommand or request, and handle/control input and output signalsaccording to the command or request. In addition, processor 26preferably operates independently of host computer 16 by reading sensorsignals and calculating appropriate forces from those sensor signals,time signals, and force processes selected in accordance with a hostcommand. Suitable microprocessors for use as local processor 26 includethe MC68HC711E9 by Motorola and the PIC16C74 by Microchip, for example.Local processor 26 can include one microprocessor chip, or multipleprocessors and/or co-processor chips. In other embodiments, processor 26can include a digital signal processor (DSP) chip, or state machines,logic gates, an ASIC, etc.

Local memory 27, such as RAM and/or ROM, may be coupled to localprocessor 26 in interface device 14 to store instructions for processor26 and store temporary and other data. A local clock 29 may be providedin the interface device 14 and may be coupled to the local processor 26to provide timing data, similar to system clock 18 of host computer 12.

The host computer 14 and/or local processor 26 may be used to controlthe output of haptic feedback to the user. Many embodiments aredescribed in greater detail in U.S. Pat. Nos. 5,734,373; 6,211,861;5,889,670; 5,691,898; and in copending patent application Ser. Nos.09/687,744 and 09/637,513, all of which are incorporated herein byreference in their entireties. For example, in a host-controlembodiment, host computer 12 can provide low-level haptic commands(e.g., streaming data) over bus 24, and the local processor 26 thendirectly provides the haptic commands to one or more actuators 30 toallow the data to control the actuators directly (or, the host commandsare provided to the actuators 30 by other electronics, and no processor26 is included in device 14). In another local control version, hostcomputer system 12 can provide high level supervisory commands to localprocessor 26 over bus 24, and the local processor 26 parses or decodesthe commands and manages low level control loops to sensors 28 andactuators 30 in accordance with the high level commands and forceprocesses (e.g., firmware). A mixture of these types of control can alsobe used.

Sensors 28 detect the position, motion, and/or other characteristics ofa user object 34 and/or a user's manipulation of the user object 34along one or more degrees of freedom and provide one or more signals toprocessor 26 including information representative of thosecharacteristics. One or more sensors 28 may be provided for a singledegree of freedom or multiple degrees of freedom along which the userobject 34 may be moved or forced. Examples of sensors suitable forseveral embodiments described herein are digital optical encoders,analog potentiometers, magnetic (Hall effect) sensors, stain gauges,optical sensors, velocity or acceleration sensors, or other types ofsensors. Either relative or absolute sensors can be used. Sensors 28 canprovide an electrical signal to an optional sensor interface 36, whichcan be used to convert sensor signals to signals that can be interpretedby the processor 26 and/or host computer system 12. Alternately,processor 26 can perform these interface functions without the need fora separate sensor interface 36 or the sensor signals can be provideddirectly to host computer system 12.

Actuators 30 transmit forces to the user 22 manipulating the interfacedevice 34, where the forces can be transmitted via object 34 and/orthrough another feature of the interface device, such as the housing, inresponse to signals received from local processor 26 and/or hostcomputer 14. If forces are output to the user object, the actuators canoutput forces in one or more directions along one or more degrees offreedom of object 34; an actuator 30 can provided for each degree offreedom along which forces are desired to be transmitted. In tactileembodiments where forces are transmitted to the user not kinestheticallyvia the user object, the actuator(s) 30 can cause forces in the housing,user object 34, or other contacted surface; such forces can includepulses, vibrations, textures, etc. Some embodiments can include drivetransmissions (gears, capstans, belt drives, etc.) to amplify forceoutput. Some embodiments can include an actuator assembly to convertactuator output to a force having a desired direction, magnitude, etc.For example, U.S. application Ser. No. 09/585,741, incorporated hereinby reference in its entirety, describes one such assembly.

Actuators 30 can include two types: active actuators and passiveactuators. Active actuators include motors, linear current controlmotors, stepper motors, pneumatic/hydraulic active actuators, voice coilactuators, moving magnet actuators, torquers, and other types ofactuators that transmit a force to move or force an object. For example,active actuators can drive a rotational shaft about an axis in a rotarydegree of freedom, or drive a linear shaft along a linear degree offreedom. Passive actuators can also be used for actuators 30. Thepassive actuator may comprise a magnetic particle brake, a frictionbrake, a pneumatic/hydraulic passive actuator, or the like, and may beused in addition to or instead of a motor to generate a dampingresistance or friction in a degree of motion. Actuator interface 38 canbe optionally connected between actuators 30 and processor 26 to convertsignals from processor 26 into signals appropriate to drive actuators30.

Other input devices 39 can optionally be included in interface device 14and send input signals to local processor 26. Such input devices caninclude buttons, dials, switches, or other mechanisms. The interfacedevice may have a separate power supply 40 or may be powered through itsconnection to the host computer 12. Alternatively, power from the hostcan be stored and regulated by interface device 14 and thus used whenneeded to drive actuators 30. Safety switch 41 may be included ininterface device 14 to provide a mechanism to allow a user 22 tooverride and deactivate one or more actuators 30, or require a user 22to activate one or more actuators 30, for safety reasons.

The user object (or manipulandum) 34 may be a device or article that maybe grasped or otherwise physically contacted by a user 22 and which isin communication with interface device 14. The user 22 can manipulateand move the object along provided degrees of freedom to interface withthe graphical environment generated by the host application program theuser is viewing on display device 20. The user object 34 may be, forexample, a joystick, mouse, trackball, stylus, steering wheel, medicalinstrument (laparoscope, catheter, etc.), pool cue, hand grip, knob,button, or other article.

Many types of user interface devices can be used with the features ofthe inventions described herein. Some examples of user interface devicesare described in U.S. Pat. Nos. 6,100,874; 6,166,723; 5,767,839;6,104,382; 6,154,201; 5,790,108; and 5,805,140, all of which areincorporated herein by reference in their entireties.

The system of the present invention may comprise two primary modes or“control paradigms” of operation for interface device 14: positioncontrol (isotonic) mode and rate control (isometric) mode. In positioncontrol mode, movement of the interface device is directly translatedinto motion of a controlled cursor or entity, i.e. a direct mappingbetween user object and controlled object is provided. Thus, motion of amouse on a surface is mapped to equivalent motion of a controlled cursordisplayed on the display screen in the same direction and for aproportional distance. In rate control mode, movement of the interfacedevice is translated to a new rate of change of a controlled value or acharacteristic of a controlled object based on the new interface deviceposition. For example, motion of a joystick in a direction can be mappedto motion of a controlled cursor in an equivalent direction at avelocity proportional to the position of the joystick with reference tothe joystick's origin position. Under rate control, the user object canbe held steady at a given position but the simulated object undercontrol can be in motion at a given commanded velocity, while positioncontrol only allows the object under control to be in motion if the userobject is in motion.

FIG. 2 is a diagrammatic illustration of a display screen 20 displayinga graphical environment. In the version shown, the graphical environmentis a graphical user interface (GUI) 200 used for interfacing with anapplication program and/or operating system implemented by computer 12.One embodiment described herein implements haptic feedback technologiesin position control (i.e. isotonic) mode to embellish a graphical userinterface with physical sensations. Alternatively, an isometric mode maybe employed, as described in U.S. Pat. No. 5,825,308, which isincorporated herein by reference in its entirety. By communicating withinterface device 14, the computer 12 can present not only standardvisual and auditory information to the user, but also physical forces.These physical forces can be carefully designed to enhance manualperformance as described below. A detailed explanation of forces andforce effects provided within a GUI or other graphical environment isdisclosed in U.S. Pat. No. 6,219,032, incorporated by reference hereinin its entirety. A detailed explanation of various interface devicesusable for navigating the graphic image in the graphical environment aredisclosed in U.S. Pat. Nos. 5,825,308; 6,219,032; 6,078,308, 6,088,019,6,243,078; 6,211,861; and U.S. patent application Ser. No. 09/585,741,all of which are incorporated herein by reference in their entireties.

Herein, the manual tasks of the user to move a cursor displayed onscreen 20 to a desired location or displayed object in a graphicalenvironment by physically manipulating user object 34 in order tonavigate the graphical environment, are described as “targeting”activities. “Targets,” as referenced herein, are defined regions in thegraphical environment, such as the GUI 200, to which a graphical object,such as a cursor, may be moved by the user. The “target” may beassociated with one or more forces or haptic effects, for example one ormore forces or haptic effects may be associated with a graphical objectof the GUI 200 graphical environment. In the GUI 200 version, targetscan be associated with, for example, graphical objects such as icons,pull-down menu items, and buttons. A target usually is defined as theexact dimensions of its associated graphical object, and is superimposedand “attached” to its associated graphical object such that the targethas a constant spatial position with respect to the graphical object. Inthe GUI context, “graphical objects” are those images appearing on thedisplay screen which the user may select with a cursor to implement afunction of an application program or operating system, such asdisplaying images, executing an application program, or performinganother computer function. For simplicity, the term “target” may referto the graphical object itself; thus, an icon or window itself is oftenreferred to herein as a “target.” However, more generally, a target neednot follow the exact dimensions of the graphical object associated withthe target. For example, a target can be defined as either the exactdisplayed area of an associated graphical object, or a target can bedefined as only a portion of the graphical object. A target can also bea different size and/or shape than its associated graphical object,and/or may be positioned a distance away from its associated graphicalobject. The entire screen or background of GUI 200 can also beconsidered a “target” which may provide forces on user object 34. Inaddition, a single graphical object can have multiple targets associatedtherewith.

Upon moving the cursor to the desired target, the user typicallymaintains the cursor at the acquired target while providing a “commandgesture” associated with a physical action such as pressing a button,squeezing a trigger, or otherwise providing a command to execute aparticular program function associated with the target. The commandgesture can be provided from any input device. For example, the “click”(press) of a physical button positioned on a mouse while the cursor ison an icon allows an application program that is associated with theicon to execute. Likewise, the click of a button while the cursor is ona portion of a window allows the user to move or “drag” the windowacross the screen by moving the user object. The command gesture can beused to modify forces or for other functions in the present invention aswell. Or, the command gesture can be provided by manipulating thephysical object of the interface device within designated degrees offreedom and/or with graphical objects displayed on the screen. In otherembodiments, graphical objects on the screen can provide a commandgesture when manipulated by a user. For example, a spring force on userobject 34 can be associated with pressing a graphical button with acursor to provide the feel of a mechanical button.

The GUI 200 permits the user to access various functions implemented byan operating system or application program running on computer system12. These functions typically include, but are not limited to,peripheral input/output functions (such as writing or reading data todisk or another peripheral), selecting and running application programsand other programs that are independent of the operating system,selecting or managing programs and data in memory, viewing/displayfunctions (such as scrolling a document in a window, displaying and/ormoving a cursor or icon across the screen, displaying or moving awindow, displaying menu titles and selections, etc.), and otherfunctions implemented by computer system 12. For simplicity ofdiscussion, the functions of application programs such as wordprocessors, spreadsheets, CAD programs, video games, web pages, andother applications as well as functions of operating systems such asWindows™, MacOS™, and Unix, will be subsumed into the term “programfunctions.” Typically, application programs make use of such functionsto interface with the user; for example, a word processor will implementa window function of an operating system (or GUI, if the GUI is separatefrom the operating system) to display a text file in a window on thedisplay screen. In addition, other types of interfaces are similar toGUI's and can be used with the present invention. For example, a usercan set up a “web page” on the World Wide Web which is implemented by aremote computer or server. The remote computer is connected to hostcomputer 12 over a network such as the Internet and the Web page can beaccessed by different users through the network. The page can includegraphical objects similar to the graphical objects of a GUI, such asicons, pull-down menus, etc., as well as other graphical objects, suchas “links” that access a different web page or page portion whenselected. These graphical objects can have forces associated with themto assist in selecting objects or functions and informing the user ofthe graphical layout on the screen. Such an embodiment is described ingreater detail in U.S. Pat. No. 5,956,484, which is incorporated hereinby reference in its entirety.

GUI 200 is preferably implemented on host computer 12 and processorusing program instructions. The use of program instructions to performfunctions and operations on a host computer and microprocessor is wellknown to those skilled in the art, and can be stored on a “computerreadable medium.” Herein, such a medium includes by way of examplememory such as RAM and ROM coupled to host computer 12, memory, magneticdisks (diskette, hard disk, etc.), magnetic tape, optically readablemedia such as CD ROMs, semiconductor memory such as PCMCIA cards or gamecartridges, etc.

In FIG. 2, the display screen 20 displays GUI 200, which can, forexample, be implemented by a Microsoft Windows® operating system, aMacintosh operating system, X-Windows in Unix, or any other availableoperating system incorporating a GUI. In the example shown, a window 201contains various icons 202 that are grouped by window 201, here labeledas “Main”, “Startup”, and “Tools”, although other or different icons maybe grouped within window 201. A menu bar 204 may be included in window201 in some GUI embodiments which permits pull-down menus to appear byselecting menu heading targets 205 with a user-controlled graphicalimage, such as a cursor 206, that is controlled by the user via a userobject 34. In the subsequent description, the terms “user-controlledgraphical image” and “cursor” will be used interchangeably.

The present invention provides haptic feedback to the user through userobject 34 based on a location, a velocity, an acceleration, a history ofone or more of these values, and/or other characteristics of the cursor206 or other graphical objects within the GUI 200 environment (positionof the cursor herein is generally associated with the position of userobject 34, unless a rate control embodiment is described). Other“events” within the GUI may also provide forces. Several preferredembodiments of different forces or haptic effects can be output to theuser, some of which are described in U.S. Pat. Nos. 6,219,032 and6,211,861. These haptic effects can be forces of a single magnitude inone direction, or they may be an interaction or sequence of forces, forexample, to create the sensation of a texture, a vibration, a dampingforce, a spring, a barrier, etc. The terms “force”, “force sensation”,“haptic”, and “haptic effect” are used interchangeably herein.

Many different types of force sensations, or “force effects” as referredto herein, can be output to the user based on interactions of the cursorwith graphical objects in the GUI 200. Two types of haptic feedbackeffects can benefit from the present inventions: kinesthetic hapticeffects and tactile haptic effects. Kinesthetic effects are those thatare output by outputting forces on the user object 34 in at least one ofthe degrees of freedom of the user object. For example, a joystickhandle rotatable along X- and Y-axes may have motors to output forces inthose axes, allowing kinesthetic haptic effects on the joystick handle.A kinesthetic mouse device allows forces to be output in the planardegrees of freedom of the mouse. A tactile haptic effect typicallyoutputs forces to the user not the degrees of freedom of the userobject; for example, a vibration can be output to the user contactingthe housing of the device. Some haptic effects, such as vibrations, canbe output by kinesthetic haptic devices or tactile haptic devices.

One type of haptic effect is an attractive/repulsive force. In oneembodiment, targets such as window 201, icons 202 and menu headings 205can have attractive force fields associated with them to bias the userobject 34 toward the target and enhance the user's ability to movecursor 206 to or around the targets. Thus, this type of haptic effect issuitable for kinesthetic haptic feedback devices in which forces can beoutput on the user object 34, in the degrees of freedom in which theuser object 34 is moved to control the cursor or other graphical image.In one embodiment, the haptic feedback depends upon a distance betweencursor 206 and a target, such as window 201. The distance can bemeasured from one or more points within the window 201 or its perimeter.

For example, icons 202 may have an attractive force associated withthem. This attractive force can originate from a desired point I withineach icon 202, which may be located at the center position of the icon,or located at a different area of icon 202, such as near the perimeterof the icon. Likewise, window 201 may have an attractive forceassociated with it which originates from a point W within window 201,which may be at the center of the window. Points I and W are consideredto be “field origin points.” Alternatively, force fields can originatefrom a point or region not shown on the screen. These attractive forcesare known as “external forces” since they affect the cursor 206 when thecursor is positioned externally to the targets. External and internalforces of targets are described in greater detail with respect to FIG.3.

The attractive forces associated with window 201 and icons 202 areapplied to user object 34 to influence the movement of user object 34and cursor 206. Thus, an attractive force associated with window 201will cause host computer 12 (or processor 26) to command the actuators30 of interface device 14 to apply appropriate forces on user object 34to move or bias the user object 34. Forces are applied to user object 34in a direction such that cursor 206 is correspondingly biased in adirection toward field origin point W of window 201. It should be notedthat the forces to user object 34 do not actually have to move the userobject in the appropriate direction. For example, when using passiveactuators, the user object cannot be physically moved by the actuators.In this case, resistive forces can be applied so that user object 12 ismore easily moved by the user in the appropriate direction, and isblocked or feels resistance when moving in other directions away from ortangent to point W.

The attractive force applied to user object 34, which would move or biascursor 206 toward point W, is represented by dotted line 207 in FIG. 2.The force can be applied with reference to a single reference point ofcursor 206, which is the tip point T in the described embodiment. Inalternate embodiments, the reference point can be at other locations orareas. The attractive forces can be computed, for example, with a 1/R or1/R² relationship between field origin point W or I and cursor tip T tosimulate gravity.

Repulsive force fields may also be associated with a field origin point.For example, it may be desired to prevent cursor 206 from moving to oraccessing particular regions or targets on the screen within GUI 200. Ifwindow 201 is one such target, for example, a repulsive field in theopposite direction to that represented by line 207 can be associatedwith window 201 and can originate at field origin point W. The forcewould move user object 34 and cursor 206 away from the target, making itmore difficult for the user to move cursor 206 onto the target.

In one embodiment, the position of cursor 206 determines which fieldforces will affect the cursor 206 and user object 34. As described inFIG. 3, targets can be associated with internal and external forces inrelation to cursor 206. For example, attractive forces can be externalforces and thus affect user object 34 and cursor 206 only when thecursor 206 is positioned externally to the target. In one embodiment,only the external forces of the highest level targets that are externalto cursor 206 will affect the cursor 206 and object 34. Thus, in FIG. 2,only the attractive force of window 201 may affect cursor 206 and userobject 34, since the icons 202 and menu headings 205 are at a lowerlevel (being within window 201). If cursor 206 were positioned withinwindow 201, only the attractive fields of icons 202 and menu headings205 would affect cursor 206 and user object 34 and the attractive force207 would preferably be removed. In alternate embodiments, the forcesfrom various targets can be combined or excluded in different ways.

In another example of such an embodiment (not shown), multiple windows201 can be displayed on display screen 20. All three windows can be atthe same hierarchical level, so that when the cursor 206 is positionedoutside the perimeter of all three windows, cursor 206 and user object34 are influenced by a combination of the three external attractiveforces, one attractive force from each window. These attractive forcescan be summed together as vectors to provide a resulting totalattractive force in a resultant direction having a resultant magnitude.When the cursor is positioned in any of the windows, only externalforces from objects within that window can be applied.

Tactile force feedback may also be applied to user object 34, whereforces can be output through the housing or user object 34 by, forexample, moving a mass, shaking the housing, moving parts of the housingor other buttons or surfaces, etc., which the user feels by contactingthe housing or moving portion. Some examples of tactile devices aredisclosed in U.S. Pat. No. 6,211,861 and U.S. application Ser. No.09/585,741, all incorporated herein by reference in their entireties.Tactile force sensations can include vibrations, pulses or jolts,textures, etc. For example, when the user moves the cursor 206 over theborder of window 201, a pulse can be output on the housing of the userobject 34 or device 14, informing the user of this motion. Similarly,when the user moves the cursor over words in a word processing program,pulses can be output. Vibrations or textures can be output while theuser is within a certain region, or dragging an object, or resizing anobject, for example.

FIG. 3 is a diagrammatic illustration of a displayed target illustratingone embodiment of internal and external forces associated with a target.As referred to herein, “external forces” are those haptic effectsassociated with a target which affect user object 34 when the cursor 206is positioned externally to that target, i.e. when the cursor positionedoutside the perimeter of the target. These types of haptic effects mayoften include attractive or repulsive forces. In contrast, “internalforces” are those forces associated with a target which affect userobject 34 when the cursor 206 is positioned internally to the target,i.e., within the perimeter of the target. Each target can have externalforces and internal forces assigned to it. The internal forces and/orexternal forces associated with a target may be designated as zero,effectively removing those forces.

Target 220 may include an external target region 222 to which anexternal force associated with target 220 is assigned. External region222 is defined from a target point in the target to a range limit 224outside the target, where the external force will be in effect. Theexternal region can be defined from outer perimeter 226 of target 220,or from an inner perimeter 228 of the target 220, if such perimeters areimplemented. Attractive, repulsive, texture, vibration, or other hapticeffects may be assigned as external forces to targets. In otherembodiments, a “groove” external force can be provided for graphicalobjects, as described in U.S. Pat. No. 6,219,032.

The internal force associated with a target affects user object 34 onlywhen the cursor 206 is within the perimeter of the target. In someembodiments, the inner perimeter 228 is not used and the target 220 canbe associated with a single internal haptic effect that is outputwhenever the cursor 206 is positioned within the target. In otherembodiments, an internal target region may include a center region 230and a capture region 232. Center region 230 is defined as the innermost,central region of target 320 and extends to an inner perimeter 228. Inthe center region, forces associated with the center region (“centerregion forces”) applied to cursor 206 can be zero magnitude so as toallow substantially free movement of the cursor within this region(also, any external forces of any targets included within target 220would be in effect). Alternatively, a particular force or haptic effectcan be associated with center region 230, such as a texture orvibration, for example.

The capture region 232 is preferably provided at or near a perimeter oftarget 220. The forces associated with capture region 232 are applied tocursor 206 when the cursor is positioned within or is moved through thecapture region. If the sampling rate of a sensor is too slow to detectcursor 306 within the capture region, a history of sensor readings canbe checked to determine the path of the cursor and whether the captureforce should be applied to user object 12. In the preferred embodiment,two different forces can affect cursor 206, depending on whether thecursor exits target 220, or enters target 220. When the cursor is movedfrom center region 230 to external region 222, an exit effect can beapplied to user object 34. For example, the exit effect can be a barrieror “snap over” force positioned at inner perimeter 228, which preferablyincludes a spring force as represented symbolically by springs 234 inFIG. 3. The spring force causes a spring resistance to the motion ofcursor 206 in the exit direction, which starts as a small resistiveforce in the direction toward the dead region 230 and which increases asthe cursor is moved closer to outer perimeter 226. This barrier forceprevents the cursor from easily “escaping” the target 220. Other forcescan be substituted in other embodiments, such as a damping barrierforce, a pulse, a vibration, etc. Outer perimeter 226 of target 220 canin some embodiments define a snap distance (or width) of the barrier, sothat once cursor 206 is moved beyond perimeter 226, the exit captureforce is removed.

When the cursor 206 enters target 220, an entry effect can be applied touser object 34. For example, the entry effect may be the same effect asthe exit effect, in the same direction toward the dead region 230 (if adirection is provided). For instance, if the entry effect is a spring,when cursor 206 first enters the capture region, the spring force canimmediately begin to push the user object 34 in an equivalent directionas toward the center region. The closer the cursor is positioned to thecenter region, the less spring force is applied. Alternatively, an entryforce different from the exit force can be applied. In such anembodiment, the direction of movement of cursor 206 should beestablished so that it is known whether to provide the exit captureforce or the entry capture force.

Other forces can also be applied to the user object 34 (or other part ofinterface device 14) when operating interface device 14 in isotonicmode. For example, an “inertia” force can be applied when graphicalobjects are manipulated by the user for particular types of targets andwhen specific conditions are met. For example, the inertia force can beapplied to the user object when the user moves cursor 206 into centerregion 230, holds down a button on the user object, and moves or “drags”the graphical object (and associated target 220) with cursor 206 acrossscreen 20. The dragged target 220 has a simulated “mass” that willaffect the amount of inertia force applied to user object 34. In someembodiments, the inertia force can be affected by the velocity and/oracceleration of cursor 206 in addition to or instead of the simulatedmass. Other factors that may affect the magnitude of inertia force, suchas gravity, can also be simulated. Alternatively, an icon's mass can berelated to how large in terms of storage space (e.g. in bytes) itsassociated program or file is. Thus, force feedback can directly relateinformation about a target to the user. In addition, damping and/orfriction forces can be provided instead of or in addition to the inertiaforces. For example, each graphical object can be assigned a simulateddamping coefficient or a coefficient of friction. Such friction might beuseful when free-hand drawing in a CAD program, where the coefficient offriction might be based on “pen size.” A texture force might also beapplied when a graphical object is dragged. In addition, if simulatedmasses are being used to calculate the external force of a target, suchas an attractive gravity force, then that same mass can be used tocompute an inertia force for the target. Such haptic effects typicallycan be output by kinesthetic haptic feedback devices.

Also, inertia forces of graphical objects can also be applied due tocollisions or other interactions with other graphical objects andtargets. For example, if cursor 206 is dragging an icon, and the iconcollides with the edge of a window, then a collision force can beapplied to user object 34. This collision force can be based on thespeed/direction of the icon/cursor as it was moved, the simulated massof the icon and/or cursor, and any simulated compliances of theicon/cursor and the edge. Also, certain edges, objects, or regions inGUI 200 can either be designated as “pass-through” objects or as “solid”objects that provide barrier forces that do not allow the cursor to passinto the objects.

Other examples of forces and associated graphical objects and functionsinclude providing force jolts or “bumps” when the cursor 206 encountersa region, when an object is released after having been dragged acrossthe screen, when a window is entered or exited by the cursor, or when awindow is opened or closed. In a text document, these bumps can beprovided when the cursor moves between words, lines, letters,paragraphs, page breaks, etc. Forces can be associated when a button ina GUI is “pressed”, i.e., moved “into” the screen and back out, and/orwhen command gestures are provided. A “snap to” force simulates a detentin a surface, thus providing a small attraction to a point. This can beuseful for menu items or snap-to grid lines in a CAD program orconstraining motion to perpendicular or 45-degree angle directions.

Yet other forces include a spring force associated with a position of atarget before it is moved. For example, when the user drags an icon, aselection of text, or pull-down menu, a virtual spring is simulated asbeing attached between the icon's current and former position. Such aspring or other type of force can also be provided on user object 34when a graphical object is resized between former and current sizes. Forexample, if the window is dragged to a larger size, then a “stretching”spring force can be applied to the user object, and if the window isdragged to a smaller size, then a “compressing” spring force can beapplied. Such features can be provided in a CAD program when graphicalobjects are stretched or otherwise manipulated.

The forgoing concepts and preferred embodiments can also be applied toother graphical objects appearing in a GUI. For example, pull-down menus(such as a “File” pull-down menu) and menu items in the menu can provideinternal and external forces to assist a user in selecting menu items.Similarly, a scroll bar or “slider” can be associated with forces, suchthat the guide and “thumb” of the slider can be associated with externalforces and internal forces to assist the user in manipulating theslider. “Pop-up” windows and panels in GUI 200 can similarly be providedwith forces, where buttons in the pop up window may have external andinternal forces associated with them. Forces associated with buttons canbe “turned off” or otherwise changed after the button has been selectedby the user using cursor 206.

It should be noted that similar haptic feedback can be provided in othergraphical environments, such as non-GUI graphical environments. Forexample, in a 3-D video game, texture forces of a dungeon wall might befelt when a user moves a cursor over the wall. Or, a tank selected bythe user with a cursor might have a high inertia force associated withit when it is moved in comparison to a small infantry soldier.Alternatively, in a 1-D control knob, a divot or pop might be felt whena target location, such as a numerical target on a scale, is approachedby a pointer.

The present invention may provide isometric (rate control) functionalityin the same interface device that provides isotonic (position control)functionality. For example, in one embodiment, isometric mode is enteredby selecting an input device such as a button. Once isometric mode isentered, an opposing force on the user object can be applied by theactuators 30 (in kinesthetic haptic feedback devices), and the user'sinput force on the user object is provided as isometric (or elastic)input to the host computer to control, for example, the scrolling of adisplayed document, the changing of a value, the speed of a cursor, thezooming or panning of a view, etc. In another embodiment, theinteractions between a controlled graphical object such as a cursor andother graphical objects allow isometric input. In other embodiments, theisometric input can be applied to a displayed window or a differentassociated graphical object in a different way, e.g., to control a textcursor in the window, to zoom the view in the window (see FIGS. 4a-b),to pan the view in the window, etc. The whole view of screen 20 canalternately be panned or scrolled using such isometric input in someembodiments. In a similar embodiment, isometric input can be provided byinteracting cursor 206 with typically nonisometric graphical objectsdisplayed on screen 20.

FIGS. 4a and 4 b are diagrammatic illustrations of display screen 20showing an isometrically-controlled zoom function of a CAD program. FIG.4a shows a cube 270 as a graphical object as displayed by the CADprogram. The cube 270 can be manipulated as desired by the user tochange the shape of the cube or alter other characteristics. In manycases, isotonic input is the most natural and efficient type of input tomove, stretch, copy, or otherwise manipulate cube 270. A user may wishto zoom in the view of cube 270 to see additional detail. In thepreferred embodiment, this may be conveniently accomplished by providingisometric input. The view of FIG. 4b shows a zoomed-in view of a portionof cube 270, where dashed box 272 of FIG. 4a indicates the extent of thezoomed view. In this example, to zoom from the view of FIG. 4a to theview of FIG. 4b, the user can press and hold an input device such as abutton on the interface device. This can cause a computer-generatedresistive force to be applied to the user object in all directions as aresult of actuator control. The user then moves the user object againstthis force in an upward direction to cause a magnification zoom. Whenthe user releases the button, normal isotonic manipulation of cursor 206is allowed.

In a different embodiment, the user may use cursor 206 to control thezoom function. In FIG. 4a, a “zoom in” isometric object 274 and a “zoomout” isometric object 276 are displayed. Alternatively, the objects neednot be displayed. The cursor 206 can be moved against any surface of theappropriate object 274 or 276 to command the associated zoom function ofthe CAD program. The present invention also allows additionalcomputer-generated forces to be overlaid on the resistive isometricforce on user object 34. For example, when the user reaches the maximumzoom magnification, a small jolt can be applied to the user object 34 toinform the user of this condition. In different tactile embodiments ofinterface device 14, such jolts can be provided in an isometric controlmode, but resistive forces are not applied to the motion of the userobject.

Enhanced Navigation in Graphical Environments Haptic Effects as aFunction of Velocity

The navigation of a graphical environment can be enhanced to furtherimprove a user's efficiency and/or enjoyment during the navigation. Forexample, a number of graphical objects, each with a associated hapticeffect, may be positioned between a cursor and a target. As a usermanipulates the user object 34 to position the cursor at the target,several of the untargeted graphical objects may be in the pathway of thecursor, so that the user may cause the cursor to pass over theuntargeted graphical objects. This can result in the user experiencinghaptic effects for untargeted graphical objects that are not ofimmediate interest to the user. This overloading of haptic information,or “haptic clutter,” may be disconcerting to the user and may reduce ornegate the positive haptic effect that would otherwise be experiencedupon reaching the target or during motion to the target. To make hapticnavigation more natural, haptic sensations should only be output to theuser when they are relevant, such as, for example, when the cursor isnear the target position, or only output for objects that are intendedfor targeting.

In one version of the inventions described herein, user intention forcursor motion can be related to the velocity of motion of the cursorand/or user object of the interface device. The haptic clutterassociated with passing the cursor over untargeted graphical objects maybe reduced by relating characteristics of the haptic effect to thevelocity of the cursor. It has been discovered that a user typicallycauses the cursor to move at a high velocity when a target is not inimmediate proximity to the cursor and then slows the movement of thecursor as it approaches nearer to the target. Thus, according to oneinvention herein, when the cursor is moving at a high velocity, thehaptic effects may be lessened in strength or muted since it is unlikelythat the cursor is near an object targeted by the user. As the cursorslows, the haptic effects are strengthened to enhance the user's abilityto more accurately and quickly locate the target.

As shown in FIGS. 5a and 5 b, the strength of the haptic effect may beproportional to the velocity of the movement of the cursor. Herein, forposition control (isotonic) embodiments, the velocity of the cursor isalso intended to mean the velocity of the user object in one or moredegrees of freedom which controls the cursor. The velocity may bedetermined in different ways. For example, the velocity can bedetermined by the interface device 14 (e.g. local processor 26) or byhost computer 12 by dividing the measured motion or displacement of thecursor (or user object) by a specific time interval. For example, in oneversion, the local processor 26 may determine the position of the cursorat fixed intervals of time. The change in displacement during the periodbetween intervals may be divided by the amount of time that lapsesduring the period, and the value may be stored as the current velocityof the cursor. Also, several such velocity values can be stored overtime and averaged to determine the velocity. Alternatively, when usingperiods of fixed duration, the change in displacement is directlyrelated to an approximation of the velocity of the cursor during theperiod, and the change in displacement may be stored as the currentvelocity of the cursor. In other embodiments, the velocity can bedetermined in other ways; for example, velocity sensors can directlydetermine the velocity of the user object 34, or velocity can bedetermined from acceleration data from an accelerometer measuring motionof the user object 34.

According to one aspect of the present inventions, after a haptic effecthas been determined based on cursor interaction with a target, and thehaptic effect is ready to be output on the user object 34 (e.g., whenthe cursor passes into the external or internal region of an object, or,more generally, passes over, or within a haptic-activating area of, agraphical object having an associated haptic effect), the magnitude ofthe haptic effect is adjusted based on a function of the currentvelocity of the cursor (which may in some cases be the same as orequivalent to the current velocity of the user object 34 in at least onedegree of freedom). Once the magnitude is adjusted, the haptic effect isoutput and felt by the user (if the magnitude is more than zeromagnitude after adjustment). It should be noted that the haptic effectmagnitudes are determined in a normal fashion, and then the magnitudesare adjusted or modulated according to the present inventions.

The function that maps cursor/object velocity to haptic effect strengthcan be any function, such as linear, a higher order polynomial function,step function, discontinuous function, etc. It should be noted that whena strength of a haptic effect is referred to as “zero” herein, this canbe implemented in many different ways, including setting a haptic effectto play with a zero magnitude, revoking the command of the hapticeffect, stopping the haptic effect, etc.

For example, the graph 300 of FIG. 5a shows a linear proportionalmapping of cursor velocity to magnitude (strength, i.e. the force gain)of a haptic effect. The force gain is normalized to a range of zero toone. At the lowest cursor velocities at or near zero, the strength ofthe haptic effect has a gain of “1” and at the highest velocities, thegaine is close to “0.” Between velocities of zero and the highestvelocity measurable, the strength of the haptic effect is determinedbased on predetermined linear curve 302, where higher cursor velocitiesmap to a lower strength effect. Thus, the faster the user moves the userobject 34, the more it is assumed that he or she does not want to selectnearby graphical objects, and the haptic sensations caused by cursorinteractions with those nearby objects are reduced in magnitude. Thelocal processor can perform this haptic effect magnitude change in someembodiments or modes, while the host computer can perform this functionin other embodiments or modes.

In the graph 306 of FIG. 5b, a different embodiment shows a stepfunction 308 used for the mapping between effect strength andcursor/user object velocity. At a predetermined range 310 of lowvelocities from zero to a predetermined velocity threshold V1, thestrength of the haptic effect is a first value, for example 100%magnitude, a gain of “1.” In a predetermined range 312 of highvelocities, from V1 to the highest measured velocity, the strength ofthe haptic effect is multiplied by a gain of K1 which is less than one.This embodiment is simpler to perform but does not provide the gradualadjustment of effect strength that the embodiment of FIG. 5a does.Alternatively, the strength gain function of FIG. 5b may include moresteps than the single step shown in FIG. 5b to achieve a higherresolution effect strength modification based on velocity.

FIG. 5c is a graph 320 illustrating another embodiment of a functionthat can be used to map cursor or user object velocity to haptic effectstrength. For a first section 322 of the described function, at lowvelocities in the range from zero to a first velocity threshold V1, thegain is at 1, so that the haptic effect strength is normal. In the rangeof velocities between threshold V1 and a higher threshold V2, thefunction 324 is used, which linearly maps velocities to magnitude usinga linear function. In section 326 of the function, where the range ofvelocities is above threshold V2, the gain is set to a value K2, whichis gain less than 1 and at the level of the low end of the linearfunction 324. This function thus provides “saturation” regions at thehighest and lowest velocities. Velocity thresholds V1 and V2 can bechosen based on which velocities users typically move the user object 34at when desiring full force strength and minimal force strength. Inother embodiments, additional function sections can be included, ordiscontinuous functions.

By adjusting the strength of a haptic effect in accordance with cursorvelocity, the use of beneficial haptic effects can be increased. Forexample, in a word processing application, a haptic effect associatedwith every character in a document can be overwhelming for the user,since the cursor can move over many characters in a short space of time.However, by setting a haptic effect strength so that the haptic effectsassociated with characters are only felt when the cursor is movingslowly over the characters (or are felt at much less magnitude at fastercursor velocities), the user will be given improved feedback on thepositioning of the cursor within the document. This is because the userwould only typically need to experience the character-associated hapticsensations when moving the cursor slowly on or around those charactersto manipulate or navigate through them (for placement of a text cursor,etc.). Similarly, in spreadsheet applications, haptic effects associatedwith individual cells, or with information within cells, may be outputat or near full strength only when the cursor is moving at lowervelocities, when the user wishes to manipulate data at the detail levelof individual cells.

Some advantageous uses of velocity-modulated strength of haptic effectscan be, for example, in selecting graphical objects or items in agraphical environment, in cursor placement within the graphicalenvironment, and in dragging and other similar tasks. When performingselecting tasks, the user may typically reduce the speed of cursormotion to be able to finely move the cursor to a particular graphicalobject and select it, e.g. text or a cell in a spreadsheet. Since cursorspeed is slow for such a task, the stronger haptic sensations output forthe slower speeds can aid in selection. Cursor placement tasks alsobenefit from the modulated haptic effects. When placing a cursor betweentwo words, two characters, or two cells in a spreadsheet, for example,the user moves the cursor slowly, and can benefit from stronger hapticsensations to aid in the cursor placement, e.g. haptic output such aspulses or vibrations informs the user where precisely objects arelocated in related to the cursor. For example, a pulse that is output ateach character haptically informs the user precisely where the cursor iswithin a word. Finally, dragging and other “analog tasks” which move,modify, or otherwise manipulate graphical objects can benefit fromvelocity-modified haptic sensations. For example, resizing an object,moving an object, rotating an object, etc., can benefit from hapticeffects informing the user of the particular task being performed oraiding in the object manipulation (with textures and/or damping, forexample, to make fine motions easier).

Some embodiments can examine the velocity of the cursor in differentways for use in modulating the strength of the haptic effects. Forexample, the average velocity of the cursor over a predetermined periodof time can be examined. Some embodiments may use an absolute cursorvelocity in the graphical environment, while others may use the velocityof the cursor between graphical objects, e.g. how fast the cursor ismoving between one text word and the next, between one text characterand the next, or between one icon and the next encountered icon.

In some embodiments, more than one velocity function may be used. Forexample, one function can be used at one range of velocities, while adifferent function can be used for a different range of velocities.Another embodiment can, for example, cater magnitude strength todifferent motions in different directions or situations. For example, ithas been determined that users may have different vertical cursormovement characteristics than horizontal movement characteristics. Afirst user may use, for example, a horizontal tool bar displayed at theupper portion of a display screen and accordingly would develop a highervelocity routine in navigating the cursor to targets at the top of thescreen. Another user may have a vertical toolbar on the side of a screenand accordingly develop a routine of fast horizontal movements. Inaddition, within an application such as a word processing application ora spreadsheet, vertical and horizontal cursor movements are often atdifferent rates. Thus, the strength of a haptic effect may be a functionof a vertical velocity of a cursor and a function of a horizontalvelocity of the cursor, where different functions can be associated withhorizontal and vertical velocities. For example, considering thevelocity threshold (V₁) version of FIG. 5b, there may be provided avertical velocity threshold above which haptic effects are output at alower strength and a different horizontal velocity threshold above whichhaptic effects are output at a lower strength. In another version, alinear or other function of vertical velocity may be used to determine astrength component and a linear or other function of horizontal velocitymay be used to determine a second strength component. The strengthcomponents may be summed or otherwise related or combined to determinethe strength of the haptic effect to be output to the user object 34.The multiple velocity functions may be predetermined, may beindividually selected by a user, or may be scaled so that a user needonly select one velocity function, for example the user may select ascaled value and the vertical velocity threshold and the horizontalvelocity threshold are automatically determined and applied duringnavigation. Alternatively, the velocity function(s) used can be adaptiveto user motion, e.g. determined on the fly by analyzing user motion fora period of time. Functions may also be determined based on otherfactors or characteristics. For example, average user velocities, pastuser velocities, areas on the screen where the cursor often ispositioned, the extent of the screen which the user moves the cursor to,the application program being used, etc., may each or all be useful indetermining a function that is best suited to the user and/or to theprogram or environment being used.

In another version, different velocity functions may be associated withdifferent haptic effects and/or with different targets. For example, ifa haptic effect of a vibration is associated with an icon and a hapticeffect of a pop or pulse is associated with a character in a wordprocessing application, different velocity thresholds may be desirablefor modulating the strengths of the respective effects. A user may beable to target an icon at a higher velocity than the user can target anarea between two characters. Accordingly, the icon effect velocitythreshold may desirably be greater than the character effect velocitythreshold. As a result, above a upper velocity threshold, both hapticeffects would be output at a low strength, such as a gain of “0”, andbelow a lower velocity threshold, both haptic effects would be output ata higher strength, such as at a gain of “1”, and at an intermediatevelocity between the upper and lower thresholds, the character effectmay be output at a low strength and the icon effect may be output at ahigher strength.

In some embodiments, different thresholds or functions can be used fordifferent targeting tasks in the graphical environment. For example, inone embodiment, for a selection task as described above, a functionhaving a lower velocity threshold can be used, allowing greater strengthhaptic effects to be output for more velocities. Since, in manyembodiments, the user is pressing a button or performing some othercommand gesture during selection, it is known whether selection istaking place and thus haptic effects can be output more strongly. With acursor placement task, a function having a higher velocity threshold canbe used to allow only very slow movements to cause full strength hapticeffects, since it may not be known whether the user desires to place acursor with fine motion, or is simply moving the cursor around withcoarse motion (in the latter case, haptic effects may not be desired).Dragging objects, resizing objects, and other “analog tasks” can usedifferent functions than other tasks, if desired.

The use of velocity as a modulator for haptic effects and, in effect, tohelp determine the user's volition in cursor navigation, is advantageouswhen selecting and highlighting text or cells in word processing orspreadsheet applications. For example, when it is desirable to highlightonly a few words in a word processing application or a few cells in aspreadsheet program, strong haptic effects are desirable to enhance theuser's ability to accurately select the intended words or cells.However, when highlighting several pages of text or several rows orcolumns of cells, it is less necessary to have haptic feedback for finecursor movements. Since the cursor is typically moved at a highervelocity when highlighting large areas, the velocity filters of FIGS.5a, 5 b and 5 c and those discussed above are advantageous.

Object Density for Use in Modulating Effect Strength

Other characteristics of the graphical environment may also be used toadjust haptic effect strength. For example, the cursor may encounter alarge number of graphical objects when moving across the screen even ifthe cursor velocity is low, if there is a high density of graphicalobjects. For instance, small characters in a word or a large number ofsmall icons on a crowded desktop screen may undesirably cause lots ofhaptic effects to be output if the user moves the cursor relativelyslowly across all these objects. Therefore, velocity alone may not insome circumstances provide a good indication of when to reduce strengthof haptic effects associated with objects, since the same cursorvelocity may in one case call for higher strength effects, while indifferent case of higher numbers of graphical objects, or moreclosely-spaced graphical objects, call for lower strength hapticeffects. In addition, in some embodiments, velocity determination may betoo coarse, e.g. velocity may not be able to be measured with sufficientaccuracy to provide helpful haptic effects.

One embodiment of the present inventions can look at the rate that thecursor encounters or interacts with graphical objects (e.g. moves overor within the haptic-activating area), which is indicative of thedensity of the graphical objects, to help determine whether hapticeffect strength should be reduced to aid navigation of the graphicalenvironment. For example, as the cursor is moved across severalgraphical objects, a count of the objects can be made within apredetermined period of time. If it is determined that a large number ofgraphical objects (e.g. over a predetermined threshold number ofobjects) are being encountered by the cursor in that period of time, thehaptic effect strength can be reduced. The strength reduction can beperformed according to a function similar to those functions of FIGS.5a-5 c or others discussed above, but having a horizontal axis based onthe count of graphical objects encountered by the cursor rather thanvelocity of the cursor, where a higher object count reduces the hapticeffect strength. Other methods can also be used to determine objectdensity. For example, the time elapsed when the cursor is moved betweentwo graphical objects can be measured for modulating haptic effectstrength. Once a cursor leaves one object, a timer measures the timeuntil the cursor enters another object; the length of time can determineeffect strength modulation according to a predetermined function. Insome embodiments, short time durations between objects indicates highobject density and may cause haptic effect strength to be reduced sothat the user is not overwhelmed with many haptic effects all at once;longer times between objects may allow the haptic effects to be outputat high or full strength.

These graphical object density determinations can also be made inconjunction with the velocity determination referred to for theembodiments of FIGS. 5a-5 c and similar embodiments, where bothgraphical object density and cursor velocity can be used to find themost appropriate strength for haptic effects. For example, if a lowvelocity is measured but a high object density is found, the hapticeffect strength can be reduced, and if a high velocity is measured, thehaptic effect strength can be reduced without even having to look atgraphical object density. The adaptive functions and other featuresdescribed above can also be used in conjunction object densitydetermination where appropriate.

Haptic Effects as a Function of Engagement Duration

Navigation of a graphic image in a graphical environment mayalternatively or additionally be enhanced by outputting a haptic effecthaving a strength based on or that is a function of the time duration ofcursor engagement (i.e., interaction) with a target. For example, atarget that has an associated haptic effect, such as an attractive orspring force, can assist the user in positioning the cursor over thetarget when the user is navigating the cursor to the target. However,that haptic effect (or any haptic effect output upon exit of the cursorfrom the target) may discourage or fight against movement of the cursoraway from the target when the user is moving the cursor to a differenttarget. It has been determined that the exiting haptic effect can beenhanced by making the strength or duration of the effect a function ofthe amount of time the cursor is engaged with the target (e.g.,positioned over the target). In one version, the strength of the hapticeffect is at least partially inversely proportional to the amount oftime. Accordingly, when a cursor is positioned over a target for alonger amount of time, a smaller-strength haptic effect will be outputwhen the user directs the cursor away from the target.

In one embodiment of the invention, a time threshold (t₁) ispredetermined. FIG. 6a is a graph 350 illustrating one embodiment ofproviding a time threshold, where the horizontal axis indicates the timethe cursor is at or engaged with the target, and the vertical axisindicates the strength of the associated haptic effect(s), or normalizedforce gain. Function 352 indicates that at those amounts of time belowthe time threshold t1, an exiting haptic effect is output at a firststrength, here shown as a gain of “1” (normal strength), and at thosetimes above the time threshold t1, the exiting haptic effect can beoutput at a second strength, here shown as a gain k1 that is less than“1,” for example “0” to turn off the haptic effect entirely, or anyfraction of 1 to reduce the strength of the effect. Thus, the hapticeffect does not interfere with the user moving the cursor away from thetarget, since time t1 is close enough to the time of engagement so thatthe haptic effect is turned off or reduced in strength typically beforethe user can move the cursor away. The amount of time at a target may bedetermined by the host computer 12 or by the local processor 26. Timethreshold t1, in one example, can be about 100-200 ms, but may bedifferent time spans in different embodiments or for different graphicalobjects, tasks, haptic effects, etc. Alternatively, the function shownin FIG. 6a may be provided with additional steps or may be a linear orhigher order function.

FIG. 6b is a graph 360 illustrating a different function 362 for usewith engagement duration. The effect strength rises rapidly at low timeamounts, tapers off at peak time P, and then gradually drops to lowereffect strengths at larger time amounts. This version is advantageous inthat exiting haptic effects are not as reduced in strength soon afterthe cursor engages the target, but only gradually reduces effectstrength, thereby reducing the amount of overshooting of a target duringnavigation. Accordingly, a user can direct a cursor toward a target withless precision and with less concentration than if the user mustcarefully position the cursor over the target and stop the cursorthereon.

In some embodiments, the timer that starts the counting of time ofengagement can start only after any haptic effects output upon initialengagement have subsided. For example, when a cursor first encounters atarget, a haptic effect may be output, such as a pulse or vibration. Itmay in some cases be a more compelling experience to the user for themethod to wait until the initial output effect has finished playingbefore starting the count of time of engagement, even though this may beafter the actual time of engagement. For haptic effects that have noduration, such as an attractive force that always attracts the cursor toa target, the timer can begin counting immediately upon engagement ofthe target by the cursor.

In other embodiments, the strength of the associated haptic effect(s)can be based on other durations instead of cursor engagement duration.For example, one embodiment can provide haptic effect strength as afunction of the amount of time the cursor is not moving (e.g. afterbeing positioned at a target).

Using engagement time (and times of other cursor activity) to modulatethe strength of a haptic effect is also valuable for haptic effects thatare applied within a target. For example, if a vibration is applied toindicate positioning over a target, modulating the effect strength overtime will lessen or remove the haptic effect when it is not needed. Thismodulation also allows for the application of additional haptic effectswithin the target without overloading the user with different effects.

As with the velocity-dependent effects, the time-dependent effects mayhave multiple time-dependent functions. For example, one target may havea first time threshold and a second target may have a second timethreshold; or a single target may have different time thresholds and/orfunctions, dynamically determined based on past cursor movement by theuser or other events or characteristics of the graphical environment.Or, different cursor navigation tasks can use different functions, e.g.for selecting, placement, dragging, etc.

Although the present invention has been described in considerable detailwith regard to certain preferred versions thereof, other versions arepossible, as discussed above. Thus, alterations, permutations andequivalents will become apparent to those skilled in the art upon areading of the specification and study of the drawings. Also, thevarious features of the embodiments herein can be combined in variousways to provide additional embodiments of the present invention.Furthermore, certain terminology has been used for the purposes ofdescriptive clarity, and not to limit the present invention. Therefore,the appended claims should not be limited to the description of thepreferred versions contained herein and should include all suchalterations, permutations, and equivalents as fall within the truespirit and scope of the present invention.

What is claimed is:
 1. A method, comprising determining a velocity of acursor in a graphical environment; and outputting a signal for renderinga haptic effect, a magnitude of the haptic effect related to thevelocity of the cursor by an inverse function.
 2. The method as recitedin claim 1 further comprising adjusting the magnitude of the hapticeffect by a constant gain, if the velocity of the cursor is less than athreshold value.
 3. A method as recited in claim 1 further comprisingmoving a graphical object in the graphical environment using the cursor.4. A method as recited in claim 1 further comprising rotating agraphical object in the graphical environment using the cursor.
 5. Amethod as recited in claim 1 further comprising re-sizing a graphicalobject in the graphical environment using the cursor.
 6. A method asrecited in claim 1 manipulating the cursor with a user-manipulableobject.
 7. A method as recited in claim 6 further comprising outputtingthe haptic effect to said user-manipulable object.
 8. A method,comprising: determining a density of objects encountered by a cursor ina graphical environment; outputting a signal for rendering a hapticeffect, a magnitude of the haptic effect related to the density ofobjects.
 9. A method as recited in claim 8 further comprising selectinga predetermined function to relate the magnitude of the haptic effect tothe density of objects.
 10. A method as recited in claim 9 wherein thepredetermined function includes that the magnitude of the haptic effectincreases as the density of objects decreases.
 11. A method as recitedin claim 10 wherein the predetermined function is one of an inversefunction and a step function.
 12. A method as recited in claim 8 furthercomprising determining a velocity of the cursor in the graphicalenvironment, and adjusting the magnitude of the haptic effect based onthe velocity determination.
 13. A method as recited claim 8 wherein thedetermination of the density of objects includes counting a number ofgraphical objects encountered by the cursor in a predetermined period oftime.
 14. A method as recited claim 8 wherein the determination of thedensity of objects is based on a time elapsed between the cursor exitingone graphical object and entering another graphical object.
 15. Amethod, comprising: determining an engagement duration of a cursor witha graphical object in a graphical environment; and outputting a signalfor rendering a haptic effect, a magnitude of the haptic effectdecreasing with the engagement duration.
 16. A method as recited inclaim 15 wherein the magnitude of the haptic effect is related to theengagement duration by one of an inverse function and a step function.17. A computer-readable medium on which is encoded computer program codecomprising: program code to determine a velocity of a cursor in agraphical environment; and program code to output a signal for renderinga haptic effect, a magnitude of the haptic effect related to thevelocity of the cursor by in inverse function.
 18. A computer-readablemedium on which is encoded computer program code comprising: programcode to determine a density of objects encountered by a cursor in agraphical environment; and program code to output a signal for renderinga haptic effect, a magnitude of the haptic effect related to the densityof objects.
 19. A computer-redable medium as recited in claim 18 furthercomprising program code to select a predetermined function to relate themagnitude of the haptic effect to the density of objects.
 20. Acomputer-readable medium as recited in claim 19 wherein thepredetermined function includes that the magnitude of the haptic effectincreases as the density of objects decreases.
 21. A computer-readablemedium as recited in claim 20 wherein the predetermined function is oneof an inverse function and a step function.
 22. A computer-readablemedium as recited in claim 18, further comprising: program code todetermine a velocity of the cursor in the graphical environment; andprogram code to adjust the magnitude of the haptic effect based on thevelocity determination.
 23. An apparatus, comprising: a user-manipulableobject, capable of manipulating a cursor in a graphical environment; ahaptic-effect actuator, coupled to said user-manipulable object; aprocessor, in communication with the haptic-effect actuator; and amemory storing code to be executable by said processor, including: codeto determine a velocity of the cursor in the graphical environment; andcode to output a haptic-rendering signal; said haptic-effect actuatorconfigured to receive the haptic-rendering signal and output a hapticeffect to the user-manipulable object, a magnitude of the haptic effectrelated to the velocity of the cursor by an inverse function.
 24. Anapparatus, comprising: a user-manipulable object; a haptic-effectactuator, coupled to the user-manipulable object; a processor, incommunication with said haptic-effect actuator; and a memory storingcode to be executable by said processor, including: code to determinedensity a density of objects encountered by a cursor in a graphicalenvironment; and code to output a haptic-rendering signal; saidhaptic-effect actuator configured to receive the haptic-rendering signaland output a haptic effect to the user-manipulable object, a magnitudeof the haptic effect related to the density of objects.
 25. An apparatusas recited claim 24 wherein said memory further comprises code to selecta predetermined function to relate the magnitude of the haptic effect tothe density of objects.
 26. An apparatus as recited claim 25 wherein thepredetermined function entails that the magnitude of the haptic effectincreases as the density of objects decreases.
 27. A method, comprisingdetermining a velocity of a cursor in a graphical environment; andoutputting a signal for rendering a haptic effect, a magnitude of thehaptic effect related to the velocity of the cursor by a step function.